Method and apparatus for providing picture privacy in video

ABSTRACT

The present invention relates to a method and apparatus for providing picture privacy in video. The method includes separating, by the processor, pixels of a first type from pixels of a second type in a current gray-scale frame based on a decision threshold, where each of the pixels of the first and second types includes an image pixel value, and applying, by the processor, a privacy filter to the pixels of the first type. The privacy filter is configured to randomly swap the image pixel values within a pixel area to occur at different locations in the pixel area. The method further includes combining, by the processor, the filter pixels of the first type with the pixels of the second type.

PRIORITY INFORMATION

This application claims priority to Provisional Application No.61/321,232 filed Apr. 6, 2010; the entire contents are herebyincorporated by reference in their entirety.

BACKGROUND

Background privacy filters suppress the background in video applicationssuch as video teleconferences, for example. For instance, a user wantsto be seen on a video conference call, but may not want backgroundinformation such as personal or proprietary items to be seen by theother users. Similarly, foreground privacy filters suppress theforeground in video applications such as video monitoring, for example.For instance, some countries have legal restrictions on videomonitoring, or some people have issues of privacy in a public or privatespace. In this case, the people being subjected to the video monitoringwould want to have their image suppressed. Therefore, a foregroundprivacy filter would suppress the person's image while allowing thebackground to be viewed.

Conventional methods for background/foreground suppression sometimesproduce errors where foreground is erroneously detected as backgroundand vice versa. Conventional methods employed to date typically haveerror correction methods, but errors can rarely be reduced to zero.Therefore, the result of background/foreground suppression is often notgood enough because an image with foreground/background holes in thevideo is considered low quality and reflects poorly on the product. Oneparticular example uses a blurring filter in which background-detectedpixels are replaced by a low-pass filtered (blurred) result of pixels atthe same location from the current video frame. However, even thoughthis may be an effective approach at background removal, it results inthe video looking “cheap” (i.e., we are used to blurred images beingpoor quality). This also requires a large blurring filter tosufficiently obscure the background, and large blurring filters arecomputationally expensive.

In addition, conventional methods for separating the background pixelsfrom the foreground pixels suffer from technical problems. For instance,after a difference between a current frame and calculated backgroundframe is obtained, the difference image is subjected to a thresholdingoperation. The conventional art uses a manual static thresholddetermination that does not adapt for lighting changes and backgroundupdates. More sophisticated methods of automatically thresholding thedifference image employ traditional statistical methods such asminimization of in-class error to determine the background andforeground pixel classes. However, these traditional statistical methodsare not optimized to images and often leave holes in what shouldcorrectly be uniform foreground or background regions. As a result, thebackground and foreground pixels may contain a significant amount oferrors. Thus, a more effective approach is desired.

SUMMARY

The present invention relates to a method and apparatus for providingpicture privacy in video.

The method includes separating, by the processor, pixels of a first typefrom pixels of a second type in a current gray-scale frame based on adecision threshold, where each of the pixels of the first and secondtypes includes an image pixel value, and applying, by the processor, aprivacy filter to the pixels of the first type. The privacy filter isconfigured to randomly swap the image pixel values within a pixel areato occur at different locations in the pixel area. The method furtherincludes combining, by the processor, the filter pixels of the firsttype with the pixels of the second type.

In one embodiment, the first type is background pixels and the secondtype is foreground pixels. In an alternative embodiment, the first typeis foreground pixels and the second type is background pixels.

The separating step may further include determining a difference imagebased on a difference between the current gray-scale frame and aninitial image, calculating image connectivity for the difference image,and obtaining the decision threshold based on the calculated imageconnectivity. The separating step may further include determining anumber of threshold levels for the difference image, where the obtainingstep obtains the decision threshold based on a local maximum of anaverage run-length over the number of threshold levels. The separatingstep further includes determining the initial image based on at leastone lateral movement of a user.

According to embodiments of the present invention, pixels having imagepixel values below the decision threshold are pixels of the first typeand pixels having pixel values above the decision threshold are pixelsof the second type. The method may further include determining whetherto apply the privacy filter.

According to embodiments of the present invention, an average intensityand color of an image that includes the combined pixels is same as anoriginal image of the combined pixels.

The system includes a video camera configured to capture video data anda processor configured to separate pixels of a first type from pixels ofa second type in a current gray-scale frame of the video data based on adecision threshold, where each of the pixels of the first and secondtypes includes an image pixel value. Further, the processor isconfigured to apply a privacy filter to the pixels of the first type,where the privacy filter is configured to randomly swap the image pixelvalues within a pixel area to occur at different locations in the pixelarea. Still further, the processor is configured to combine the filterpixels of the first type with the pixels of the second type to create aresulting video image. The system also includes a video displayconfigured to display the resulting video image.

According to one embodiment, the first type is background pixels and thesecond type is foreground pixels. According to an alternativeembodiment, the first type is foreground pixels and the second type isbackground pixels.

Also, the processor is configured to determine a difference image basedon a difference between the current gray-scale frame and an initialimage, to calculate image connectivity for the difference image, and toobtain the decision threshold based on the calculated imageconnectivity. The processor is configured to determine a number ofthreshold levels for the difference image, where the processor obtainsthe decision threshold based on a local maximum of an average run-lengthover the number of threshold levels. The processor is configured todetermine the initial image based on at least one lateral movement of auser.

According to embodiments of the present invention, pixels having imagepixel values below the decision threshold are pixels of the first typeand pixels having pixel values above the decision threshold are pixelsof the second type. The processor is configured to determine whether toapply the privacy filter.

According to embodiments of the present invention, an average intensityand color of an image that includes the combined pixels is same as anoriginal image of the combined pixels.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments will become more fully understood from the detaileddescription given herein below and the accompanying drawings, whereinlike elements are represented by like reference numerals, which aregiven by way of illustration only and thus are not limiting of thepresent invention, and wherein:

FIG. 1 illustrates an apparatus for providing picture privacy in videoaccording to an embodiment of the present invention;

FIG. 2 illustrates a method for providing picture privacy in video inwhich foreground is designated “Fg” and background is designated “Bg”,according to an embodiment of the present invention;

FIG. 3 illustrates a method for separating pixels of a first type frompixels of a second type according to an embodiment of the presentinvention;

FIG. 4( a) illustrates a difference image according to an embodiment ofthe present invention;

FIG. 4( b) illustrates a graph showing a decision threshold for thedifference image of FIG. 4( a) according to an embodiment of the presentinvention; and

FIG. 5 illustrates a privacy filter according to an embodiment of thepresent invention.

DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

Various embodiments of the present invention will now be described morefully with reference to the accompanying drawings. Like elements on thedrawings are labeled by like reference numerals.

As used herein, the singular forms “a”, “an”, and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”,“comprising,”, “includes” and/or “including”, when used herein, specifythe presence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The present invention will now be described with reference to theattached figures. Various structures, systems and devices areschematically depicted in the drawings for purposes of explanation onlyand so as not to obscure the present invention with details that arewell known to those skilled in the art. Nevertheless, the attacheddrawings are included to describe and explain illustrative examples ofthe present invention. The words and phrases used herein should beunderstood and interpreted to have a meaning consistent with theunderstanding of those words and phrases by those skilled in therelevant art. To the extent that a term or phrase is intended to have aspecial meaning, i.e., a meaning other than that understood by skilledartisans, such a special definition will be expressly set forth in thespecification that directly and unequivocally provides the specialdefinition for the term or phrase.

The embodiments described below illustrate a foreground/backgroundsuppression technique that includes a separation technique (e.g.,separating background pixels from foreground pixels) and a “frostedwindow” filter that is applied to the separated background or foregroundpixels. The separation technique is based on spatial connectivity of thepixels as opposed to treating each pixel with spatial independence. Thevideo foreground/background privacy filter de-emphasizes error bymaintaining intensity, color, and frequency characteristics. Thebackground/foreground suppression technique may be implemented in anytype of video camera system including mobile communication devices andvideo monitoring systems, for example.

FIG. 1 illustrates an apparatus for providing picture privacy in videoaccording to an embodiment of the present invention. For example, theapparatus may include a video camera 101, at least one processor 102,memory 103, and a video display 104. The video camera 101 may be anytype of component capable of capturing video data during a videoconferencing session or video monitoring application, for example. Thememory 103 may be any type of memory capable of storing at least aportion of the video data received by the video camera 101. The videodisplay 104 may be any type of output component that is capable ofdisplaying video data such as a liquid crystal display (LCD), forexample. The processor 102 is configured to control at least the videocamera 101, the memory 103, and the video display 104. These componentsmay be embodied in one device, or two or more devices according to anytype of combination. Also, the apparatus of FIG. 1 may include othercomponents that are well known in video camera systems.

FIG. 2 illustrates a method for providing picture privacy in videoaccording to an embodiment of the present invention. After the videodata is captured by the video camera 101, in step S210, the processor102 determines whether to apply a privacy filter to the captured videodata. For example, when the privacy filter is applied, the processor 102suppresses either the background or the foreground. For instance, whenthe privacy filter is implemented in a mobile communication device, thecamera phone may provide three options: full image, backgroundsuppressed, or foreground suppressed. If the “full image” option isselected, the privacy filter is not applied to the video data. Rather,the video data is left unaltered and allowed to be displayed without thebackground or foreground being suppressed. Also, the foregroundsuppression option may be implemented in a video monitoring device. Forexample, the processor 102 may apply the foreground privacy filter basedon the density of people in an area, the number of pedestrians walkingpast the camera, the number of people before an interactive advertisingsign, or the existence of a public telepresence station, for example.However, if the “background suppression” option or the “foregroundsuppression” option is determined to be applied, the method continues tostep S220.

In step S220, the processor 102 separates the background pixels from theforeground pixels based on a decision threshold. The decision thresholdis the threshold that separates one type of pixels from another type ofpixels. In one embodiment, the decision threshold may be an optimalthreshold. For instance, the threshold that separates the backgroundpixels from the foreground pixels is optimized by using a connectivitycriterion function, as explained with reference to FIG. 3. However, itis noted that although the description of the embodiments describe thevideo data as two levels (background and foreground), the video data maycontain any number of levels.

FIG. 3 illustrates a method for separating the background pixels fromthe foreground pixels (or vice versa). In S310, the processor 102calculates an initial image (background or foreground) based on previousgray-scale frames stored in the memory 103, as explained below. Forexample, if background suppression is used, the initial image is aninitial background image. If foreground suppression is used, the initialimage is an initial foreground image. For instance, in regards to videomonitoring, the initial foreground image may be determined based onprevious gray-scale frames stored in the memory 103. In regards to videoconferencing, the initial background image may be determined as furtherexplained below.

For example, the processor 102 registers the initial image in a mannerthat allows the user to stay in the image while the background isregistering. For instance, the user first signals that in the next fewseconds or so, the user wishes to initiate the background image whilethe user remains in the image. The background initialization processbegins and the user is instructed to move in a rightward and leftwarddirection. For instance, the processor 102 may instruct the user to leanan inch or so to the left and an inch or so to the right. However, anylateral movement will suffice. Then, the processor 102 determines theleft and right boundaries of the user, and joins the left and rightboundaries to obtain a silhouette of the user. The processor 102 usesthis silhouette to modify the initial background image by removing theuser's image (e.g., the foreground) from this background image, andsetting those pixels to a value that signifies unknown background pixelvalues. The resultant “hole” of unknown background pixel values issubsequently updated through the regular background updating process asmore frames are captured, and if/when the user moves away from that areato reveal the true background. The rest of the specification explainsthe privacy filter in terms of background suppression. This is done forexplanatory purposes only. However, it is recognized that the backgroundprivacy filter explained below would equally apply to the foregroundprivacy filter.

Once this initial background image is determined, in S320, the processor102 determines a difference image according to a difference between theinitial background image and the current gray-scale frame. In S330, theprocessor 102 applies a thresholding operation to the difference imagein order to obtain a number of thresholds. Thresholding is an imageprocessing operation applied to the difference image (gray-scale videodata) to obtain binary intensity level images or multiple intensitylevel images. A gray-scale difference image typically has a large rangeof intensity values, e.g., 256 values, but usually less levels ofinformation. Proper thresholding requires both proper selection of thenumber of thresholds needed to represent the number of levels ofinformation and also the selection of the optimum values for thesethresholds. Embodiments of the present invention employ the thresholdingoperation as explained with reference to U.S. Pat. No. 5,781,658, whichis incorporated by reference in its entirety.

After the number of threshold levels is determined, in S340, theprocessor calculates image connectivity in order to obtain the decisionthreshold in step S350. For instance, image connectivity can beapproximated by x and y run lengths at each threshold level of thedifference image, T_(i), as shown below in equation (1).

$\begin{matrix}{{{C\left( T_{i} \right)} = {\frac{1}{N_{r}}\left\lbrack {{R_{x}\left( T_{i} \right)} + {R_{y}\left( T_{i} \right)}} \right\rbrack}},{0 \leq T_{i} \leq T_{\max}}} & {{Equation}\mspace{14mu}(1)}\end{matrix}$

N_(r) is the number of run-lengths, and the run-lengths are calculatedaccording to Equations (2) and (3).

$\begin{matrix}{{R_{x}\left( T_{i} \right)} = {\sum\limits_{y = 0}^{Y}{\sum\limits_{x = 0}^{X}\left( {x_{E} - x_{S}} \right)}}} & {{Equation}\mspace{14mu}(2)} \\{{R_{y}\left( T_{i} \right)} = {\sum\limits_{y = 0}^{Y}{\sum\limits_{x = 0}^{X}\left( {y_{E} - y_{S}} \right)}}} & {{Equation}\mspace{14mu}(3)}\end{matrix}$

Variables x_(S) and x_(E) are the start and end of runs along rows whoseindices range from 0 to X, and variables y_(S) and y_(E) are the startand end of runs along columns whose indices range from 0 to Y. A run isa group of adjacent pixels of the same or similar intensity valuesidentified within the difference image. For example, a run may be aportion of a row or column or may be a region which is specified by aparticular area of the difference image.

In step S350, the processor 102 obtains the decision threshold based onthe local maximum average run-length determined over all possiblethresholds [0,255] of the difference image.

FIG. 4( a) illustrates the difference image according to an embodimentof the present invention, and FIG. 4( b) illustrates a graph(connectivity plot) showing the decision threshold for the differenceimage of FIG. 4( a) according to an embodiment of the present invention.FIG. 4( a) shows three lines (A, B, and C), which are plotted on thegraph of FIG. 4( b). Referring to FIG. 4( b), difference values for thedifference image of FIG. 4( a) are illustrated on the independent axis,and the average run-lengths are illustrated on the dependent axis as afunction of the difference values. Line A on FIG. 4( a) shows ahorizontal run in the background region of the difference image, whichhas the longest average run-length on FIG. 4( b) and difference valuesequal to zero. This is the peak on the connectivity plot of FIG. 4( b).Line B on FIG. 4( a) shows a horizontal run through a long, non-noisyforeground region of the difference image, which has a long (but not thelongest) average run-length and a local peak on the connectivity plot atB. The local peak is the decision threshold value. Line C on FIG. 4( a)shows a horizontal run that is within region C on FIG. 4( b). Forinstance, in region C, the connectivity plot decreases from the localpeak at B as the difference values increase. This corresponds torun-lengths that become shorter and broken as the threshold differencevalue increases.

Next, in step S360, the processor 102 separates the background pixelsfrom the foreground pixels based on the decision threshold level. Pixelswhose values are below the decision threshold are considered backgroundand pixels whose values are above the threshold are consideredforeground.

Referring back to FIG. 2, in step S230, the processor 102 applies aprivacy filter to the separated background pixels. According to anembodiment of the present invention, the privacy filter is afrosted-window filter that randomly swaps the locations of the imagepixel values within the bounds of a filter mask (e.g., a pixel area).For 2example, an image pixel value associated with one location isswapped with another image pixel value associated with a differentlocation. This “swapping” occurs for each pixel value in the pixel area,which is defined by the filter mask. Equation (3) illustrates anembodiment of the frosted-window filter.I′(x,y)=F

I=I(x−i,y−j),i,jεRandom{+−W/2}, where 0<i,j,m,n<W(filter size)  Equation(4)

The symbol,

, refers to a convolution operation of filter F upon image I. The filterF has size W×W, and in this window, pixel values of the original image,I(x,y), are swapped randomly to occur at different locations in theresultant image, I′(x,y). FIG. 5 illustrates the frosted-window filterbeing applied to a pixel area of the background pixels. Image 300 a isthe image before the privacy filter is applied, and image 300 b is theimage after the privacy filter is applied. As shown in image 300 b, theimage pixels are randomly swapped creating a “frosted-window”, which hasthe same average intensity and color as the original background pixels.

Referring back to FIG. 2, in step S240, the processor 102 combines thefiltered background pixels with the unaltered foreground pixels tocreate the final video data, which is displayed on the video display104. The above described method also applies to “foregroundsuppression.” In that case, the privacy filter is applied to theforeground pixels, and then the filtered foreground pixels are combinedwith the unaltered background pixels.

Variations of the example embodiments of the present invention are notto be regarded as a departure from the spirit and scope of the exampleembodiments of the invention, and all such variations as would beapparent to one skilled in the art are intended to be included withinthe scope of this invention.

What is claimed:
 1. A method of providing picture privacy in video databy a processor, the method comprising: separating, by the processor,pixels of a first type from pixels of a second type in a currentgray-scale frame based on a decision threshold, each of the pixels ofthe first and second types including an image pixel value; applying, bythe processor, a privacy filter to the pixels of the first type, theprivacy filter being configured to randomly swap the image pixel valueswithin a pixel area to occur at different locations in the pixel area;combining, by the processor, the filter pixels of the first type withthe pixels of the second type.
 2. The method of claim 1, wherein thefirst type is background pixels and the second type is foregroundpixels.
 3. The method of claim 1, wherein the first type is foregroundpixels and the second type is background pixels.
 4. The method of claim1, wherein the separating step includes: determining a difference imagebased on a difference between the current gray-scale frame and aninitial image; calculating image connectivity for the difference image;and obtaining the decision threshold based on the calculated imageconnectivity.
 5. The method of claim 4, wherein the separating stepfurther includes: determining a number of threshold levels for thedifference image, wherein the obtaining step obtains the decisionthreshold based on a local maximum of an average run-length over thenumber of threshold levels.
 6. The method of claim 4, wherein theseparating step further includes: determining the initial image based onat least one lateral movement of a user.
 7. The method of claim 1,wherein pixels having image pixel values below the decision thresholdare pixels of the first type and pixels having pixel values above thedecision threshold are pixels of the second type.
 8. The method of claim1, further comprising: determining whether to apply the privacy filter.9. The method of claim 1, wherein an average intensity and color of animage that includes the combined pixels is same as an original image ofthe combined pixels.
 10. A system for providing picture privacy invideo, the system comprising: a video camera configured to capture videodata; a processor configured to separate pixels of a first type frompixels of a second type in a current gray-scale frame of the video databased on a decision threshold, each of the pixels of the first andsecond types including an image pixel value, the processor configured toapply a privacy filter to the pixels of the first type, the privacyfilter being configured to randomly swap the image pixel values within apixel area to occur at different locations in the pixel area, theprocessor configured to combine the filter pixels of the first type withthe pixels of the second type to create a resulting video image; and avideo display configured to display the resulting video image.
 11. Thesystem of claim 10, wherein the first type is background pixels and thesecond type is foreground pixels.
 12. The system of claim 10, whereinthe first type is foreground pixels and the second type is backgroundpixels.
 13. The system of claim 10, wherein the processor is configuredto determine a difference image based on a difference between thecurrent gray-scale frame and an initial image, the processor isconfigured to calculate image connectivity for the difference image, andthe processor is configured to obtain the decision threshold based onthe calculated image connectivity.
 14. The system of claim 13, whereinthe processor is configured to determine a number of threshold levelsfor the difference image, wherein the processor obtains the decisionthreshold based on a local maximum of an average run-length over thenumber of threshold levels.
 15. The system of claim 13, wherein theprocessor is configured to determine the initial image based on at leastone lateral movement of a user.
 16. The system of claim 10, whereinpixels having image pixel values below the decision threshold are pixelsof the first type and pixels having pixel values above the decisionthreshold are pixels of the second type.
 17. The system of claim 10,wherein the processor is configured to determine whether to apply theprivacy filter.
 18. The system of claim 10, wherein an average intensityand color of an image that includes the combined pixels is same as anoriginal image of the combined pixels.